With the widespread adoption of the internet, a world-wide computer information network, computers from locations around the world are logically connected. Many other distinct networks exist in private companies, educational institutions, and government agencies. A useful feature of the internet is a well known service called TELNET. TELNET allows a first computer to be controlled by a second computer from virtually anywhere on their shared network. In the case of the internet, the controlling computer is often thousands of miles from the controlled computer. This allows users access to remote computers around the world.
TELNET is based on a textual user interface, typically based on the UNIX operating system. Text based user interfaces easily translate across computer environments because text, for the most part, looks the same on different operating platforms.
With the introduction of the Apple Macintosh, XWindows, and Microsoft Windows, graphical user interfaces (GUIs) have become popular. GUIs are computer user interfaces that are pictorial rather than text based. GUI environments tend to use a pointing device, like a mouse, in addition to a keyboard. Instead of typing a textual command, the user of a graphical interface typically selects a button or menu selection with a pointing device such as a mouse and "clicks" on his selection. Although GUIs are popular with a large class of computer users, GUI based programs cannot be remotely controlled with text-based TELNET.
GUI is a term of art. It is an acronym for "graphical user interface" and is used to picture and graphic based computer systems contrasted with text based computing environment. Examples of GUI's is Microsoft Windows, the Apple Macintosh, and XWindows. Examples of non GUI, text based computing environments is MS-DOS, and UNIX.
Hypertext is also a term of art. Hypertext describes a naming convention in a networking computer environment where an information source, such as an encyclopedia article, displayed in a textual format, can contain hypertext links that reference other articles or even other reference sources that in general are stored in a remote location. In modern computer systems, when a hypertext link is selected, the computer system can retrieve the information associated with the hypertext links automatically. A modern, well-known hypertext language is the hypertext markup language used by the world-wide-web of the internet, better known by its acronym HTML 6,7!.
To address the problem of controlling GUI based programs, specialized remote control programs have been created. The current GUI remote control programs generally require that the controlling machine and the controlled machine have similar screen characteristics and run identical operating environments. For example some products require both the controlling machine and the controlled machine to run similar versions of Microsoft Windows and additionally require that special remote control software to be loaded on both the controlling machine and the controlled machine. Such products cannot be used in situation where the user is not permitted to load software on a candidate remote computer.
A common component of remote control programs for GUI-based machines is a software routine that reads the GUI screen image a suitable screen data structure by some means for further processing. Such routines are widely available for most GUIS and programming environments.
Hypertext markup languages, such as HTML have been defined to store documents with visual highlights such as large type, bold type, underlining, and so forth, in a machine independent format. HTML is a well known hypertext markup language used extensively in the World-Wide-Web (WWW) service of the internet. Hypertext browser programs are written for a large number of different operating environments, thus allowing the same hypertext document to be read on very different computing environments such as an Apple MAC, an IBM Personal Computer using Microsoft Windows, or a Unix machine running XWindows. Each machine presents the document in its local environment in the best way it can. A hallmark feature of hypertext languages is a hypertext link. A hypertext link is typically a highlighted textual indicator, that, when selected by a user reading the containing hypertext document on a computer, causes a new hypertext document referenced by the link to be automatically loaded. Thus a user of a hypertext browser can access data encoded by its author in an appropriate hypertext markup language (HTML) in a very convenient and powerful way. When the text of a current document makes reference to a related topic, the user simply clicks on the hypertext link associated with the related document, and the system automatically retrieves the related document from a location generally unknown to the user, and displays it on the screen.
One of the first popular hypertext browser was for the world-wide-web (WWW) of the internet is MOSAIC. There are currently a large number of other hypertext browsers, including Cello, Lynx, Hotjava, Netscape, Spyglass Enhanced Mosaic, Mac Mosaic, Air Mosaic, Omniweb, O'Reilly Mosaic, MACS-W3. Internet MCI, Ventani Mosaic, CERN Line Mode, Pipe Web, Midas WWW, Charlotte, Digital Mosaic, NEC MosaicView, AT&T Enhanced Mosaic, NOV*IX Mosaic, Internet Commander, Quarterdeck Mosaic, WebExplorer, MacWeb, and IWENG (America On-Line). The World Wide Web (WWW) service of the internet forms a world-wide conglomeration of hypertext documents and is the fastest growing segment of the fast growing internet. Thus a large number of existing computers have a hypertext browser program already loaded on them. Hypertext browser programs and the related HTML language allow machine independent graphical presentation of data.
Using current technology, it is not possible to control GUI based machines in a general way from remote computers, either similar or dissimilar, using existing internet services. What is needed is the ability to control GUI programs remotely from a dissimilar computer, using readily available internet services and programs.
An object of this invention is to provide a means for a program based on a graphical user interface (GUI) on a first computer to be remotely controlled from a second controlling computer where the two computers may be connected directly or indirectly by means of a local area network or a wide area network such as the internet. A further object of this invention is to allow the second computer to be dissimilar from the first computer both in its user interface and its underlying operating environment. A further object of this invention is to utilize readily available, and widely used software on candidate second computers, so that the remote control functionality can be introduced solely by installing remote control software on the first computer without any installation of remote control hardware or software resources on any potential controlling computers.
An extensive body of non-patent art teaches a number of topics that are helpful, but not essential, for practicing the preferred embodiment of the present invention. This non-patent art is publicly available as copyrighted books or CD-ROMS, commercially available at bookstores and other retailers. Since these works are generally covered by copyrights of others, direct incorporation of these works here is not appropriate. In order to assist those desiring to practice the invention to better understand the existing background technology needed to practice the present invention, the following non-patent prior art copyrighted sources are hereby incorporated by reference:
1. The C Graphics Handbook, by Roger T Stevens (Academic Press, 1992), pages 731-768, teaches how to convert bitmap data into the GIF file format.
2. Graphics Programming Solutions, by Julio Sanchez and Maria P. Canton (McGraw Hill, 1995), Chapter 10.1, teaches about the GIF file format and how to read and write GIF files.
3. "Microsoft Developers Network Development Library", (Microsoft, April, 1995), Device-Independent Bitmap (DIB) API, function CopyScreenToDIB( ), teaches how to read a Microsoft Windows GUI screen into a DIB data structure; also teaches how to write a bitmap data structure into a data file. A technical article by Kyle Marsh, "Microsoft Windows Hooks", teaches how to programmatically control a Microsoft Windows program, including how to programmatically operate the keyboard and mouse using the WH.sub.-- JOURNALPLAYBACK message and functions SetWindowsHooksEx( ) and UnhookWindowsHookEx( ).
4. "Microsoft Windows Software Developer's Kit (SDK)", including Visual C++ 1.52 Help resource (Microsoft, 1995).
5. OSBORNE WINDOWS PROGRAMMING SERIES, Volume 2, General Purpose API Functions", by Herbert Schildt, Chris H. Pappas, and William H. Murray, III (Osborne Mc Graw-Hill, 1994), generally teaches the Microsoft Windows programming interface, and specifically teaches in chapter 2, 33, and 34 how to read Microsoft Windows GUI screen into a DIB data structure.
6. HTML Sourcebook, by Ian S. Graham, (John Wiley & Sons, Inc, 1995), teaches HTML and HTTP protocol, including the use of GIF files to transport WWW images to a number of well known client computing environments.
7. Web Publishing with HTML IN A WEEK, by Laura Lemay, (SAMS Publishing, 1995), Chapters 1, 4, 5, 6, 7, and 13, teaches HTML protocol, the use of GIF image files, and "clickable" images.
8. Using Netscape, by Warren Ernst (Que, 1995) teaches about Netscape, a well known hypertext browser for the world-wide-web (WWW) of the internet.
9. Building a Web Site, by net.genesis sic! and Perra Hally (Prime Publishing, 1995), Appendix B contains the complete specification for Hypertext Transfer Protocol--HTTP/1.0; and chapter 6 teaches generally about how to use HTTP in a computing environment.